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Background 


Fhis document reports on a study ‘of severdl characteris-' 
tics of the Computer Based Education (CBE) lessons written at 
military training sites whose CBE projects were sponsored by 
the. Defense Advanced Research ee aa Agency (ARPA). The 
‘ purposes of the study were: 

“53 | \ ee 
1. to demonstrate the potential of computer-based scanning 
; - of CBE programs as a means for extracting information 
about the progtanming techniques EEGs 


2. to suggest the use of computer-scanning techniques 
to aid management of CBE development efforts, and 
, re % 
3. to provide additional information about the products 
of CBE lesson development projects conducted under 
ARPA sponsorship. 


. The scope of the study was limited to include only a few 
of the more interesting and potentially useful parameters that 
can be easily measured by computer-scanning techniques. The 
lesson characteristics selected for examination were chosen 
on the basis of (a) the ease and reliability of their measure- 
ment and (b) their importance in CBE.‘ Many other characteris- 
tics could have been included, but the number studied is 
sufficient to illustrate the potential of the technique. 

Most data are reported on a site-by-site basis. The 
names of the various sites are presented, not for the purpose’ - 4 
of pointing. out strengths and deficiencies, but in order to 
allow readers who are familiar with the materials produced at ° 
a given site to relate their “intuitive perceptions of lesson 


characteristics to the actual quantities we havé measured. 


o 


; C \ 
In general, the aim of the report is to illustrate the tech- 


niques of computer-scanning and to identify important rela- 
tionships among categories of lesson characteristics. , It is 
not our intent to critique the efforts of individual sites. 

The lessons. examined here were also studied in investiga- 
tions of author language proficiency (Francis, 1976a) and the 
“use of peripheral devices connected to the terminal (Francis, 
1976b). The general technique of using the computer to scan 
or search through the source code for the presence of certain 
target "character strings" was: applied in both the study of 
language proficiency and the investigation ‘of terminal 
peripheral device’ usage. 

Although the quantities examined in this study and the 
“data collection routines used to measure them are those cere, ' 
are provided by the PLATO system, the techniques used here 
can be applied to other CBE systems which support similar | 
features. Nevertheless, we assume that most readers will | 
have some familiarity with the PLATO-oriented terminology for 
various features and measured quantities. Because documenta- 
tion of features of the PLATO system and its authoring 
language is readily available (cf. Wood, 1975: Lyman, 1977; 

Sherwood, 1977), details of these features are not presented 
here. 


. Data Collection Procedures 


The first step in the data collection process was to 
determine the nature arid number of the lessons produced at 
each of the ARPA-sponsored sites. Sites which developed only 
a small number of lessons (five or fewer ) or which poroduced 
only lesson fragments were excluded. Next, the usage data 
for each of the lessons was examined and any lesson having 
five or fewer hours of usage during the previous year was 
discarded. The latter technique insured that data storage 
files, note files, incomplete lessons, and many other "non- 
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a, 
t q + 
e, , , : 
lessons" would be:eliminated from the target group. Finally, 
.» the Tists of site-developed programs were corrected "by hand" 
so that some Wgenuine but seldom-used lessons puld be included. 
The data were collected by means of the PLATO system's 
aes "X-search" option. This feature allows the. investigator to 
specify a word, phrase, or:set of letters CLe8sy a "character 
~ string") and let the computer locate all occurrences of that, 


string. For example, if the investigator wishes to find how 


many comments (i.e., statements describing the nature and 
purpose of a piece of TUTOR coding) the author has placed in 
a program, he enters the. X-search option and specifies all 
the possible indicators of a comment. In TUTOR, the PLATO 
system's authoring language, comments can be indicated by two 
ee signs in the:"tag field" of a TUTOR’ state- 
ment, a "c" at the beginning of a line followed by a space 


consecutive 


C1, 84. Se). or by an asterisk at the beginning of a line. 
Upon locating one or more of these character strings in a 
line of theeprogram, the computer displays that line on the 
bl terminal's Screen. The investigator can then determine by 
looking at the entire line whether the occurence of "c ", say, 
is at the beginning (and hence actually indicates a comment) 
or is only a part ‘of some other coding or text. Decisions 
f of this kind can be made rather easily by clerical staff 


having minimal training. ae : 


Sites 
The ARPA authors had been taught or given suggestions ‘' 

about various lesson characteristics during their TUTOR train- 

— ing at the Computer-based Fducation Research Laboratory (CERL) 
and through consultation with the CERL staff. In most cases 
only general guidelines werd given ho the ARPA authors: for 
example, they Rete told to document any avouraand ng which was 


unusual, complex, or which interacted with coding in other 


° t ; 
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-eated.. In general, the/ local environt 
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nu val”, "complex", or "interacted 


files. The meaning on 
with" was illustrated bA 


e¥amplés, but was not further delin- 


ptyof the author was, 
more a determinant of Hie! behavior than’ Yas the training he 
received at CERL. Th 


from site to site, and 


ejenvironments did, of course, vary 
it is for this reason that a variety 
of sites were included An the study (‘see Appendix I for 
descriptions of the #i les). Furthermore, reporting the data 
ona site- by=site d Isi/s serves to illustrate oh variation in 
a a goals rand i 


Unfortunate yy 


ndards. 
it was not possible to apply exactly the 


same criteria in’! yelecting the programs/lessons to be examined 
from each site, |v cpuse of variations in site goals, methods, 


and types of mat rials. produced. Furthermore, the computer 


programs select for examination from some sites included 
non-instructign Ly routines NBs g., routers and drivers) and/or 
programs written solely ee research puE DOSES as well: as 
those written for instruction. 

Table 1 summarizes the major characteristics of the 
programs examined at each of the sites. Readers Should refer 
to Table 1 to facilitate interpretation of results presented @ 


later in the report. 


more complete identification of the sites. . 


he ogee ¢ * —_ “Table | i ae ne ee ae f an 
ae . \ Characteristics of YUTOR Programs Produced at ARPA Sités ea ne ee” 
site? Student ‘‘Tutorial = Some --. Incomplete  % ae “Res/bp? a? |  ugeaeead 
Z sted Lessons/ © Simulations Lessons © ' for Course Orientation". — Only 
CHA . \x ee aa ie No ee 100%: . Op : ® 9 Ne: 
ABE. 2X a No 100%, 5 Ope 
SHP * X X a 3 Gh a 90% : Op" ' No. - 
NTC xX , xX , No _ - 2 20% Res . | Yes 
"NI xX xX No: 100% : Ser | og. 5 aee8 
"ORL X  , | 7 No. *° “oon Aine TR Res x “Yes 
MAX X ght ge 1073s: a - Res ° Yes 
ARI x" Mes = e207 + OR Bee o> 3 No. 


& For convenience, the site names’are abbreviated throughout the nO See Appendix: I for a 


\ “xt 
b Percentage of selected programs designed to meet ae existing training need at the ‘site Ce: , 
percentage of programs which are "non- research! ot : . | 


© Objectives of sites are either research (Res) or operational (Op) in nature 


. d Lessons only - tallied as "Yes" if the programs examined were all intended: de each ide materials 
(i.e., lessons); tallied as "No" te peer also included pEaveEs) routers, and data Se ela 
routines. ; ; 


Note. All data cited in this pecker? were taken from the leseuns: as they appeared in June 1976. 
At that point nearly all sites Had- been in operation for two to four years and had established 


patterns of behavior. . 
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‘comments were found and displayed by the computer, they were 


‘tallied as comments if they contained any sort of information 


SPECIFIC CHARACTERISTICS 
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Documentation 

‘We use the term "documentation" broadly to indicate both 
the naming of variables and the commenting of lines or seg- 
ménts of programming. On the PLATO system, documentation can 
be implemented via notes off-line (on printed topies of the 
lesson or in notebooks) or via on-line comments. Because a 
poll of ARPA authors indicated little if any off-line docu- 
mentation, the study of on-line documentation herein can be: 
viewed as being essentially comprehensive. ; | . th 

The need for documentation is noha puted + Its quantity 


should be proportional to the complexity of the ding used: * 


the more complex the program, the greater the need\for docu- 
mentation. Because many of the lessons written by ARPA authors . 
do not use complex “programming, it might be expected that , " 
these lessons would not require extensive documentation. 
Nevertheless, good practice dictates that some documentation 
be provided even ee straightforward programming. 

The level of'documentation was measured by examining: 
(a) the number of on-line comments of all kinds, and (b) the 
use of defined variables. | 

On-line comments. Although comments can be placed in : 
PLATO lessons in three ways (see Appendix II), we did not Pa 
distinguish among the three comment "formats" in making our see 


counts. As lines of the TUTOR code potentially containing z 


that would he useful to the author or a later "caretaker" of 
° re cd y 
the programs. ; ‘ : 
Table 2 summarizes the rates of use of comment’s in ARPA- 


sponsored TUTOR programs. — \ \ 


MAX 7 ee tar ae ar 5. Bais . 


opment Center (NPRDC). North Island Site, (N.I.) might be so 
comphex that even an-average of 14.3 comments per aa ios is.? 
ae from the viewpoint of ‘the person charged with : i. 


vided by. the original -authors. Nevertheless, on the" basis Of 


-to find so ne, lessons with no comments t elt i ae a 


© 


Cable 2 
Use of Comments in*Prosrans Develoned at ARPA Sites 
Site y@°-  ° 4 Lessons Total # Average # 


. with Comgents Comments/ 
* Comments 3 Lesson: -. 


- CHA ~ | nS 31 0.8 


ART epee 90 313 15.5 7 . 


= 


ORL NF oo, as 57 he a 
ABE ak me tt 192 | Be 
SHE. | ws 66.08 | 2 


aon wn 


NTC Ks 15 ~ 29 12° 0, 


N.I. ¢, vig oo) ae 172 14.2. 


an = total punter of’ lessons examined at aoe ave 


TE tis difficult tosgauge ‘the adequacy of the. on-line = | a ga 
comment’ used by authors at the: various sites. ,, For example, e 


it is possible that the Chanute. pragtamming is so straight- 


forward that persons responsible for making revisions can do ; 


“$0 readily even with Baca average of .8 commented lines 
per lesson; ° On the other 


and, the nature of the programming 
in lessons produced at the Naval «Personnel Research and Devel- 


making evisions. Not having attempted to unravel, the - ee 
intrinac kes of the coding used in each program, .we- are hesi- P 


tant to pass judgment. on* the adequacy - -of the comments pro= 7 A 


our experience with inadequate documentation, | we are sorry . f 


* ¢ f 
\ : rq e ak? 
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second column of data in Table 2). Perhaps the best use. of:. k 
“the. results ‘in Table 2 is as a benchmark apacnes Which mana- i 
gers of coursew e development projects can). Ue lessons 
produced under their control. These data -will ‘be mgst useful’ -_ »* 
for those managers who are familiar with the lessons written - 
at ARPA sites -a id hence are able to judge the relevancy of 
these results for their own situation. oe fe | 

«Named variables. ‘ The computer storage locations or vari- 
ables available for use. in programs in the TUTOR language ‘can: 
be accessed in their "primitive" form (e. By N1).02 450548150" . Be, ie 
V1,V2,...,V150) or they can be given meaningful names of up = v 
to seven characters in length (e,g., nlzsindex, n2=count, oe Se, 
v3=root, etc.). Because unique names-can be assocjated with 
_the primitive variables by use of a -define- command, such 
variables are said to have been "defined". There are several 
advantages to using ‘defined: variables, but the one of greatest 
current interest is that complex uses of meaningfully-named 
' ‘variables ‘are much easier “to decipher than those using.varia- ,* 3 
bles in their primitive, Un-nared form. The use of defined | os 
variables, therefore» is an important method of documentation. ae 


a en 


All | programs selected’ from each site were searched for 
Me presence of -define- commands. Unlike "comments" which - 
ean and should be distributed SRP Onenont a program, -define- 
commands are typically used only once per lesson. A single 
-define- can be used to associate names with practically ‘any 
_number of variables. . Hence, the absence of a -define-,, command 
is ‘quite pentane because it sen neue that no named variables 


were used. 4, On-the other hand, ‘the mere: presence of a -define- : 


command does not guarantee that. all variables used had been 

* given: names. : <n of Bae ee 
_ To estimate the rate of usage of ‘defined Variables, we j 

identified eset of. ‘11 target variables and searched for ref- 


, erences to. them in “each#or. ‘the lessons (see Appendix TE for 7 


more details about the target sef). The four possible out- 


comes were that:. (a) each tarnet variaple used in the lesson. 


“had been defined. (b) no target [variables used in the lesson 


had been defined, (c) some tara, t variables used in the 
lesson had been defined and so had not, and (d) none of the 
11 target variables had been’ used’ in the lesson. Table 3 
reports the percentages at daspont at each ‘site fallinz into 
categories (a), (b), (c), afid (d); 


The right-most column of Table 4 (% of kessons with no 


-define-s) is presented so that the: reader can note the close 
congruence of those data. with. the information in the third 
“column .(% of les'sons with no- target variables ‘defined). -If 


oneyis interested only in which OF how many lessons’ ‘have no 
named variables, the data. in the right-most column of Table 3 
are far quicker to gather than those in the third column and 
hence provide essentially the same information more cheaply. 
All in all we have found the information in Table 3 to corre- 
ee well With more time-consuming personal observations. 

This approach seems adequate in most cases for making manage- 


ent decisions. For example, searching for +define- commands 
: A 
is 


a’quick way for finding Tessons which have not used named 

variables. ’ oe 
, "Nameable" pieces of a lesson awmedaieiee. to “variabies) 

include -unit- =) -entry- points, and blocks. The names ‘of 


_these pieces may be chosen py: Une ‘author, and his skill in. 
assigning them may speed later identification and recall of 


° 


their purpose. As part of this survey, we attempted to clas- 
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‘sify and categorize the patterns of authors' naming habits_ 


(e.g:, numbers, abbreviations, acronyms) + but failed to. find 
any consistent or important patterns. -Therefore, we have 


limiged our discussion of on-line: documentation to the. topics 


of use.of comments and use of. named variables. 


. zy 


b | 


10 7 
Fi 
” 
. Table 3 
+: | Defined and Primitive Variables 
s % Lessons 7, Lessons % Lessons*® % Lessons . % Lessons 
4 With All With No _ With Some With No . With No. 
Site N Target Target Target Target -Define*s 
" ' Variables Vartables Variables Variables 
‘ Defined - Defined _ Defined Observed } 
a | ne ane | a i. Oa. 
“ARI 20 95 o 0 0 > ual 
Max 7, 100 a8 a 0 0. 
ORL 16 25 31“ 38 6 26 
ABE 22 0 : 55 45 : 0 45 
GHP 66) 27 5 4h 24 7 38 
NTG*. 15° Ge 2% A00 x 0° 0 93. 
~ “es 18, 33 ae: eae: 0 34, 
e i : ; ‘ 
as 4n = total number of lessons examined at each site ae 2 
Ne ‘ ; . 
é “ly 
” # 
’ ‘ . + ‘4 ss 
Gi eh aie Ree Og lei, aes 
; ee ag oie ies: , a 
% bs a _ ee ot gel 


a? 


iY 


Courseware Engineering 
The construction features of a aap which allow it 
to be efficient and effective.in its medium constitute what 


° 


we call courseware engineering for CBE. kfficiency is con- 

sidered in terms of the extent to which the author's program~ 

ming "style" inter feres* with his lesson's use by students or 

“other authors. This concern can be thought of as a kind of = 

"human engiheering" applied to CBE courseware materials. Dis-. 

cussed 4in this section are several examples of poor courseware 
engineering habits. which we observed and measured. These 

- are: oversized lessons, poor implementation of drivers, and 

insufficient numbers of student restart points. 
Lesson size. The amount of space that a lesson occupies , ; 
is closely. related to its, usability. A large lesson, unless 
--used by ‘several students simultaneously, takes up'more than 
its "fair share" 6f active: computer memory’ (known on the PLATO 
system as*tExtended Core Storage or ECS). ‘Because ECS is allo- 
cated among the various sites attached to the system, a single. 
user of a large lesson at a given site may: prevent. other users os 
at that site” ‘from accessing their ied” ‘lessons. Large: 
lessons also are typically found to be more difficult to mod- 
a ify and troub\ eshoot. Therefore, the experienced, conscien- 
. tious. PLATO author generally tries to limit the size of his 
7 ' Lessons to approximately 3000 computer words each and divides ‘ 
'_ into two or more sections those lessons which are larger than 

' - about 5000 words.. _ &, 

_ Some lessons must, by their nature, fall outsiah Yhe a 
bounds of normal usage. However, among; the large. lessons 
produced by ARPA authors, only some of those developed at the 

_ Sheppard and San Diego sites (NTe any N.I.) were so, monolithic 
that they. Gould not be divided. . 
mm _ Lessons which ‘are too small can also ee ae ; 
for .the user’. These hazards include a greater proportion of system 


tz 


resources expended of "overhead" operations (e.g., condensing or 
disk accessing) and, if the same driver routine is used in 
each of several small lessons, a large amount of ECS may be. 
taken up by duplicate copies of the, driver's:code. Some 
programs*which present large amounts of text re utilized ° 
a small active driver routine designed to m } and display 
sections of text from a dataset file. Although such programs 
may. be effective in reducing the amount of ECS in use at any 
one time, they may require more frequent disk ‘accesses than 
can be easily supported by current system resources. -In gen-. 
eral, however, the problems associated with too-small lessgns | 
are not as troublesome as- those which: stem from excessive 
size, and no too- small ‘lessons were observed in this: study. 
Table 4 portrays several perspectives on size of lessons 
at individual sites. A few extremely large programs can be 
tolerated at a site if their impact is diluted by the use of 
many other small lessons or if large lessons are, used only. 
, during pow usage periods. ‘Therefore, to allow a few excep- 
tional lessons to be Hover iodked" in determining the range in 
lesson Size, interquartile ranges were calculated. These. are 
ranges for. the "middle" group of lessons after the largest 
and smallest 25% have been. discarded. All sizes are given in 
: terms of the number of 60-bit computer swords", of ECS occupied 
by-a- lesson while in use. . 


Only a few sites reported having shortages of ECS because of 


their use of large lessons. Chanute AFB (Dallman, DeLeo, 
Main & Gillman, 1977) and the NPRDC sites, NTC and. Nite, 
(Crawford, Hurlock, Padilla & SaSsano, 1976) reported that 

' they could not always use all their terminals because of ECS 
shortages during periods of: peak computer’ usage (10 a.m. 
PiPouee 3 p.m. CST). 
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Table 4 


* Indicators of Size of. TUTOR Programs Developed at ARPA Sites. 


Inter- 
Full quartile 
: Site N@ Mean Median ScD Range Range 
es > CHAM Rue 37 4949-4979 = 2103, 755- 342 
= , BEST 10375 6140 
. ARI 20 3544 3990.. 1691 1H5-. 2233- 
ar ay ; 6240 . 4TT4 
a - ee | | he 3 
eee MAX 7 3837 4056. -1200° «-1688- 3562- 
aes Bs ; . 5063 4372 
at. 4 S ‘ ; . ‘ ‘ , 
ORL © 16: 2492 2401 910 1246+. 4013- 
ee ; aoe 2909 
‘ . : 
ABE 22 = 3821 3869 =..1177 .. 1500- 3018- 
- " “ 6714. 4260 
“SHP 66 5055 5117 .1362 1500- / 4061- 
' 782.1 4037. 
f NTC 15 5472 = 5203 862 ‘4I7A- .5091- 
- 2 8099: 5482 
ile (a N.I. . 12° 3536 ©3691 ©2615 350- . 1764- 
1, Vo id J 8235 ‘5009 
a : a N = total number of lessons examined at each site 
: : As seen in Table 4, the full or unattenuated range shows 
Cs the CHA, NTC, and N.I. sites. to be the ones having -the largest 
iad " .single lessons. Since these also were the sites which reported 
~ the most difficulty with ECS Shortagesy it appears that the 
unattenuated range is .an adequate measure for forecasting 
Ss * problems with long’ plessons (T,25") LE probably is not neces- 
~ ' sary to compute interquartile ranges). Assuming that. future 
; - sites would have similar distributions of lesson sizes and 
use patterns, it is apparent that site managers should 
ok : : 
i ; 2 
A ae iz 19 i 
& : . # @¢ 
e ‘at : 
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" developed and successfully implemented drivers of their aa 


a | ae . ote 


direct their attention toward trying to shorten the longest 

lessons. The data suggest, but do not prove, that dividing a 

few largeslessons may sygnificantly ‘alleviate the difficulties. 
Use of drivers. In order’ to.make the most effective, use 


of authors' time, various routines known as "drivers" have been 


developed (see Frapels, 1976b, Appendix IV for an example). 

These routines aré€ used to speed the construction of commonly 
used. lesson ‘segments ‘such ,as tests, drills, data collection, 
ete. By using these pre- weather rout: nes, the author saves 

programming time when the lesson is created. . By: maintaining 

only ‘a -single copy of each routine, separated - ‘from the lessons 
employing it, the author also speeds revisions. For example,,. 
“4f the decision ‘to "time" all students in drills is made after 
: all the lessons are prepared, the authors who used a driver 


need make but one change-~-in the drifer for the drills. Authors 


who have individually coded each drill or who have copied 
standard routines ‘into eagh lesson must make. the modifications 
in each individual lesson. 

’ Two sites used drivers to rae advantage. _ Several test : 
‘drivers were developed by the MTC group and used by the ' 
Sheppard AFB. staff. Authors at the San Diego NPRDC sites a 

‘The value of drivers was well-recognized among other “ARPA 
PLATO sites, but their implementation was sometimes misunder- | 
stood, hence negating many potential benefits. For example, 
one site using MTC-supplied drivers copied the driver routines 
into EACH lesson, changing the names of units, -define=d 
variables, ‘etc. Subsequently, when modifications were required, 
. all lessons had to be individually revised with different 


. 


_variables having"to be found and corrected in each lesson. 


'. If thése routines had been. used as originally intended (i.e., 


3 w_ 2 . . : 


. 
» Py 
if a 
: “ne 
. ~ . . 
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by maintaining only a single form of the code which could be 


‘accessed. by other programs via a -use- command), it would only 


have Been necessary to make changes in that original -use-d 


', piece of coding. 


Another s¥t@ efeated drivers which ahetsved “hewy naive 
programming. A single multiple choice item, programmed 


using these drivers, congumed more than half of the student 


variables available to the programmer and several hundred . 


wofds of ECS.. ..Although the “developer of these drivers hoped. 
. ‘they would be used by “a wide PLATO audience, it is not : 


surprising that few SREHOTS made use of them. 
a. potential hazand of extensive use of driver peuviies A 
is that they may be written in a too-general format. A 
developer may be tempted’ to addea variety of effort-saving 
features which may not'be needed for all applications, hence 
increasing the overhead (in terms of ECS requirements) for 
those users who need only a subset of the driver's Aeeruress 
Thus, it is necessary to balance generality and potential 
for effort-savings of driver routines’ against the increased 
operating overhead they may require. Optimizing the design 
and application of drivers requires a degree of experience 
which few new authors will have. 

Restart “commands. -restart- commands are uSed to estab- 


lish *etentry points so that a student can begin a new session 


at an appropriate place in his current lesson. Failure to 
insert these commands can cause problems for the student: if 
he leavyés his lesson before completion or if.a system inter- 
ruption terminates his progress involuntarily, he will be 
forged ‘to, start again at the beginning of the lesson. This 
has: ‘been a source of difficulty and frustratign for students 
in the past (see. Klecka, 1977b). 

The method we have used to measure the use of the -restart- 
feature involves counting the number, of -restart- commands 


\ a ; : 


’ 


in a lesson 
of TUTOR So 


and dividing that total by the number of "blocks" 
ree code in the lesson, .a measure of the lesson's 
resulting ratio is a more refined measure than : 
estart-s per Lesson because it takes lesson size 
(i.e., it gives the flumber of -restart- commands” as 
banat size), o“ 
contains thé raw counts of -restart- commands 
various. sites. Table 6 shows the number of 


er block of source -code’. t 


‘ Table 5. 
Number of -restart- Commands. pe 


% Lessons 
with Command 


¥ Commands 


tal number of lessons examined at each site 
P 


on stude 
“lessons. 


nt jariabies to meatnnaes the aeitatiies place in the 4 


~ 


MAX 
ORL 


ABE 


* SHP 


an 


+29 
1.00 
.90 
.00 


.00 


0.01 


per Block of Source Code 


[nter- 
Full quartile 
Range Range 


0.00- -0.21- 


“f,60 * 0.64 
0.00- 0.00- 
0.44 0.10 


0.00- 0.00- 
v0 


~20 0.00 
0.00- 0.08- 
0 


59 0.36 
0.90- 0.00- 
0.00 0.00 
0.00- -0,00- 
1.80 0.24 
0.00- 0.00- 
4.50 0.00 
0.00- 0.00- 
0.03 0.00 


total number of lessons examined at each site 


. 


Although feedback from some of the sites (as* well as 


If_it is known, for example, 
n about having to repeat small 


our general experience) indicates that -restart-s were used 
less frequently than might have been desired, we do not yet 
know what ratio of restarts per block should be taken as a 
‘minimum stanfard. Hopefully, this measurement technique can 
be applied to lessons for which independent measures of 
-restart- adequacy are available. 
that students rarely complai 


18 


. 
sections of material after unplanned interruptions in-a given : : 
lesson, that. lesson's -restart-s per block ratio may be a 

reasonable standard for other lessons in that subject area 

‘and for similar populations of users. However, since’ 

measures of storage space (e.g., bloeks or units) would be 

strongly affected by authoring styles and demands of the 

manersels a more appropriate measure might be -restantés per 

display. 


Measurement of Programming and Instructional Design Sophistication 
An author's proficiency in effective and efficient use 

of the TUTOR language to program instructional lessons can 

be gauged by a use of certain commands (or language features 

such as "reserved words"). 

. Reserved words. Reserved words (aise known as system 
defined variables) are storage locations in which imformation , 
relating to a student's progress through a lesson are auto- 
matically stored by the computer. An author can write pro- 
“grams in such a way that they utilize the information in 
the reserved words to individualize the instruction or collect 


student performance data. 

{ A small group of the more than 100 reserved, words in the 
TUTOR language (Avner, 1977) was selected as a target for . 
computer-scanning of the ARPA lessons (see Appendix Il for . - 
details). These’ reserved words fell into the following four 
categories of application: ? my we 


‘ = 
1.. area data, auto-cdllected by the PLATO IV system 
to give information on student performance without 
requiring the author to maintain a count of the. ° . 
* number of questions answered correctly, the total : 
number of questions, the total number of errors i 
or requests for help, etc. 


2. time data, to record the amount of:.time taken by 
a student to complete an instruetional unit such . 
as a drill, or the amount. of time a student may 
choose to spend working practice problems. 


; . % % 
* ‘ ‘ 24 ‘ ee * ga 
. ; 
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3: feedback control information, to’ enable an author 
e-. ° to prescribe feedback messages that take account 
of the number of times a student has attempted 

to answer that question. 


4. answer judging information, to enable an author to 
eens? 7 prevent duplicate dnswers to multiple-answer ques- 
tions, etc. \ 


\ A iise of reserved words varied widely in the programs _ - 
a! ; ekanawed in this study. “Some lessons incorporated references 
to\many reserved words; each several times; others made no’ 
re rences to the reserved words selected as targéts for 
computer-scanning. “The data for all elements of the target* 
. set were pooled and. are reported in Table 7. a 4: 
* . : oe 
Table 7 | 
USage of Reserved Words 
Site. . Na . # Reserved . * % Lessons with 


F - 4 | Words Reserved Words 


' on e 
qQ ; 
. se of 7 
| 
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-Response han li ng and feedback. The, use of certain eom- 


mands and feedback\ after the judgment of ‘student: responses can 
indicate a degree o care ‘in ‘instructional design. The author 


€ 
of an instructional esson may anticipate certain incorrect 


perrren a student is \likely to give, In such cases ‘he or she 
might use a -wrong- command followed by a feedback message 
designed to correct the student's specific misundéfstanding, 
A -no- command may be used to, enable the author May POV IOS 

general feedback to unanticipated incorrect. responses: (i. e. 

| those Which do not match a -wrong- command). An author: he 7 
is less Sophisticated in response handling might ‘fail to \ e 
include either -wrong- or "os commands. ‘In such cases the 
only corrective feedhack the student: receives is the nom 
which is automatically provided by the system. Feedback | : ‘ 
delivered for specific responses could be of greater value in 
-guiding “the student toward the correct answer without - blatantly 
giving it away (see. Klecka, 1977a). In a: simitar vein, the ~ 
use of multiple or sophisticated forms of the ranswer- command 
enable an author to broaden the range of. acceptable correct | 
responses at relatively little cost. ‘The provision of instruc | 
tional "helps" via the -help- command may also serve to. indi- 
cate care in instructional design. | a 

Table 8 presents summary statistics for a variety of + 

. potential ly-useful measures of ‘the response handling ‘and | 

| feedback characteristics of lessons produced ‘at the ABE, SHP, 
and CHA sites: For purposes of interpretation it should be 
noted that the counts of -arrow- commands include any -arrow-s sie 
used for indicies and other applications which do not inyolve 
judgment of student responses. The counts shown for --wrong- add 
commands. also include instances, of -wrongve. Likewise, those i" \ 


‘given for -answer- commands include occurrences of -ansv-:and 


—-ansu-"(seeé Appendix II for details). . Ss oe 
Re \. . é i 2 
5 ead ‘ 
: ‘¢ if ‘ 
: : . ate oF, 
. 4 . , # - gt * ye? 
‘ * 7 2 26 f 
” 45 t ’ 
: t x 4 — % \ . 4 
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% | 
7 | “Tables * | 
Po a) Sh Le Measutes of Response Handling, <Beedb ck, gand 
‘ _ ‘Remedial Assistance in-SHP, ABE, and CHA? Lessons ‘ 
. Ratio © Site Mean Median S:D. Full Range - Interquartile © 
_ oe . _ Range 
sweong-/ SHED - 1.22, 0.75 2.02 0,00= 13, 50. | 0.46 - 1.40. 5. 
-arrow- | ABEG 0.90 0.88 -.0.54 0.09 -42.62-° 0,61 - 1,21 © ' 
ani, CHA: 0.78 0.28 ° 1.35, 00 -"4.00 0.00 - 0.80 , 
bieno-/' ~ SHP . 0.76 0.76 = 0.60° 0,00 = 3.62, - 0.50°=-0,94 — 
~arrow- ‘ABE 0.78. 0.8r . 0,36. .° 0.30 = 1,84 -- 0.58 = 0.87" 
c~ ’- CHA 0.36 0.39 0.26 9,06 - 0.71 0.10.- 0.46 
wo as 4 ; . Me a 
f ,(-wrong- "+ + SHP 1.98 | 1.58 2.09 0.00 -/13.50 0.70 °- 2.08 , 
: -no-)/ -. ABE 1.70 1,62 0.76 0.40 - 4.13 1,36 - 2.06 
warrowe GHA, {0.68 0.69 - 0:32 0.06; Log" 0.62 - 0,96 _ 
~answer-/  SHP * 1.23, 1,05. 1.06 ~—-0.00 = 5.80.-—0-70 - 1.38 y 
““earrow=« | ABE | 0.74 - 0.81 0,32 0.07 - 1.13 0.67 - 1.00 (3 r 
os = CHA (0.74 0.75 0.20 » 0,43 - 1.00 ' Ee - 0.88: , 
- ghelp-/ . SHP 0.10° 0:03:.° 0.18 °° 0.00,-.0;87° 0.00 - 0,13 - 
~block- ABE 0.07 0.00 0.11 0.00 - 0.39 0.00 - .0..08 
- . CHA 0.04 0.00 ~- 0.07 0.00 - 0.19 —_ - 0,00”. 
ae ‘Data for ‘the CHA lessons were collected while the lessons | were in first borer 
a ‘ , - ; 
te 7 a he e., 4 SHP Lessons were examined) ka: 
a Ca" ge a Big Ete 3 oe 
Sele \ various: ratios used as measures of dnvtructi onal: design competence ~ 


« were computed within each lesson. The summary statistics ‘reported describe 
_ the distribution of. these ratios dcross the set of -lessons. at : @ach 8 site. 
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“$ arrow= (t39=3.032, p=.005) than those from CHA. 


. Ms - | ey) 


Of the variables: measured (and reported in Table 8), 


+ there was remarkable similarity among the sites on -wrone-s/ 


. earrow= and -help-s/block in spite of marked differences in 

the PLATO. applications. ‘each employed. However, some statis-. 
tically significant differences were found among ‘the sites on 
the other measures. The ABE ,site's lessons had\more -no-s/- 


sons also had signigicantly fewer (- -wrong-+ -no-s) arrows — 
than “ABE Lessons | (t309=3. 657, =.005) ar SHP lessor ts50= =3,.883, 
p=.0003). “Finally; the ‘SHP:.lesso a significan lys 


| 7 higher -answer-/-arro atio than either. the ABE. lessons 


(6 622+838, p=.0066)-or the CHA lessqns (t5oz2. 804, ps.. 0076). 
(Probability values*for the t-statistics resulting from the’ 
last three comparisons vere estimated by Welch's correction 
‘since groups had: unequal variances). 


_ Because the .student populations ‘and instructional strate- 
gies employed: ‘at each site differed widely, i€- “Would jbe inap- 
propriate ‘to assume that these data alone. ‘indicate superiority . 


of one Set Of lessons over another. For example ,. it is: pos-. 
sible that the context in which the CHA lessons were ‘used 


~___made it: unnecessary to provide for a broad range of acceptable 


orrect responses..or diverse antic pated incorrect responses). 
«It does appear, .rowever, that ‘measu es~amenable. {0° SORES EES. 


aby ‘scanning techniques are “sensitive to differences among. “sets” 


he CHA Jes- - 
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Q : sssoris-and-that-they-are-potentially useful. to managers 
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DISCUSSION AND-CONCLUSIONS = 


- ‘ae “Specific indings 7 
, : Documentation. The lessons in the’ ‘target group of ARPA- 
‘sponsored TUTOR Programs | were not thoroughly documented a Mats 
* conclusion follows both from measureménts we have taken (see 
‘Tables 2 and 3) and. subjective judgment based on our experience 
ak ‘Sas "caretakers" of lessons following project completion. - 
Shes " ne reason for the meager levels of documentation is that 
y a most: of the ARPA PLATO" authors had no previous programming 
experience or training. The importance of documentation is 
” stressed in most formal computer science training programs, . 
-and these attitudes are reinforced in work environments. Lack- 
.ing prior experience, the ARPA authors had not previously 
‘ "internalized" the valye of, well-documented programining. Fur=, * 
thermore, many ARPA projects were considered to be short-lived 
47 "experimental" efforts whose pMograms wou d not require main- 
. ‘tenance. aheey te project had ended (hence lowering the 
, y perceived importance of documentation) . ‘ Even for. those authors 
' who did develop the habit of inserting comments in their more 
wT _ recent Phar “the burden of going “back to add comments to 
er Her-work-may-have-been-felt—as-being-too-great—to—under 
‘take’... In’ general, there were few environmental-rewards. for 
hee 3 _ including ‘ample documentation and: few punishments for not. 
sae eee Courseware 6 é po" The issue~OfLesson-stze-as tt 
‘aneene to the periodic shortages’ ‘in ECS experienced le 


‘the five year history ofthe ARPA/PLATO projects. has an 
esting parallel: to the ‘energy crisis in ‘the U. Se During 
periods when ECS ‘is: plentiful. in relation to demand, there 
_are few ‘"punishments" for writing eee large | lessons. ¥ 
Problems do occur, wever, when demand ingreases to exceed # 
; : a site's ECS ed Unfortunately, ‘the obvious remedy oe 
. of. ‘reprogramming lessons: to. _reduce their size is erten viewed i 


nter- 


: . i- ’ 
~ * r * . . 
» . :) . if 
i ‘ s at id 


ee 


oversize 


current limits. 


based on number of terminals). 
’ size of its lessons, a site gains addig§ional flexibility; and ~ 
the ability to use all of its terminals simultaneously for a 
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as too "painful" or expensive. Hence some people argue for 
conservation of resources (i.e., keeping lesson. size small). 
*while others advocate expanding resources (buying | or building 
more, memory- space). _ 


There is no doubt thet lazy programming habits wasted 


ECS, al some ARPA authors were adamant about not dividing 
essons. There. is also no doubt that there are some _ 


legitimate. needs for lessons | as large as or’ ‘larger: than the 
ae 


ineffectiveness of incentives for encouraging good program- 


_ming -habits. For example, a PLATO site with more than 12 
. terminals typically "owns" and manages its owt ECS pool (size 
By limiting (or reducing) the | — 


greater. variety.of activities. By. failing to set or enforce 


lesson size’ limits, the site is forced to let some of its 


‘terminals remain idle or to implement rigid schedules of use. 
In our experience, this: penalty had a very small effect for 
motivating staff to divide large ‘Lessons; they preferred me . 


let some terminals go unused. 


roblems with "driver" programs nated earlier ‘seem 


drivers do, how they operate, and: which resources are scarce 


plentiful. A descri tion in 


in an offcline- “manual. “could ‘indicate the, characteristics of a 
“well-constructed driver and thereby attempt to. eliminate the. 
inefficient practit¢es identified, in. this: study." ; 


The data. presented. ‘regarding the use: of ~restart-s e 


instructional lessons*(see Table 6) ‘lead to the conclusion _ 


that more. consistent usage ‘would have been desirable. One 


possible explanation for this is that -restart-s are often : 


left out until after students begin to use the lessons. - 


: a * gate 


What is most. surprising to us is: the sonarus 


largely: ‘to be the result of mistaken impressions regarding what 


"aids" or a chapter. 


e 
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‘Authors (since they are less affected by unplanned interrup- 
tions or the necessity to attend other ‘classes, etc.) may not 
‘notice the absence of -restart-s until actual students begin 
to complain. Even student tryouts of new materials may not 


-calert the author to-cases of insufficient numbers of -restart-s, 


because such trials are usually completed in one sitting. 

‘Some authors may not have added -restart-s even after lessons 
were put into use, either because;of lack of understanding of 
their importance or a belief that) adding them was a difficult 
task. We believe that these notions can be dispelled through 


“proper. training. om, 


Reserved words, response handling, and fe edbanl, ~ Commands 


“and language features discussed earlier under these headings 
can be economically described as "interaction commands". The 
/potential for extracting useful data via computer-scanning 
seems especially great for commands in this category. Because 


J of the jlarge numbers of interaction commands present in most 


lessons, their ratios tend to be stable from one lesson to 
_another (see Table Sls This stability suggests that it may 
be ssible to describe minimal standards for frequency of 
Ei aiactieh commands which can serve as guides for. new authors: 
‘or as measures of how well a given lesson is ) adapted to an - 
.interactive medium such as CBE. 
» Although this investigation measured. ratios of commands 
resent in lessons' TUTOR coding (indicative of, otential~ 
-interactions and feedback), we were not ablis- to" acquire 
~ sufficiently well-matched data to. determine ratios of those 
* commands encountered by ‘students ‘during ‘study of the lessons 
(i. Coy actual-interactions and feedback). Such "execution 
~time™ data could validate analyses of coding used in lessons 
. and aid their interpretation. Techniques for assessing the 
meaning’ and usefulness of student interaétion data are more 
“advanced at this time than those described here for program’ 


* 
t,o 
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on | 


OE Ne ! .. * 
- ee (see Francis & Weaver, 1977), “but the poten- 
p. ' tial of: the latter should be further explored and: exploited. 
b - f ‘ f : . 
Comments on Management Uses of the Computer Searching Technique 
_ This report demonstrates that several categories of data 
useful in operating a PLATO CBE site can be gathered effi- 
ciently by means of various” features of” the PLATO system itself 
(particularly its werdnaxeavelitnn capability). Levels of 
documentation and -restart- use can be monitored easily with 
these techniques, and the potential for study of interaction 
command use has been noted. This report also suggests ways 
for \minimizing the time and expense for data collection. For. 
example, it appears that monitoring the sike of only the 
largest lessons may be sufficient for avoiding problems of 


") 


long lessons and that ‘merely checking for the.presence of 
-define- ‘commands may be an ‘adequate measure of use of named 
varia les. oo =I 
© Milita, most of these ‘Sarms of data can be easily 
lected by.clerks haying a, minimal amount of-training, we . 
feel: that project managers. may ‘prefer to Re these checks ; 
_— themselves on an informal but regular basis. There is a | 


“are ‘internalized and managers are is “the best position to make 
. __ decisions about how to use the data ‘derived from. these. tech--—- 
Sanaa We” FECOGRT EE CANE WAVE TPE ENCED) UHE-UNTTTeUty———— 
of convincing authors of the need for anything that causes - . \ 
“even slightly more work in the short term when ‘payoffs are 
greatest in the long term: (for an. example, see the discussion 
o regarding documentation). We - believe, however, | that skillful je * 
i use of computer- scanning techniques can give managers an 


1 8 


‘ empirical basis for establishing guidelines and BSSESSINE 
authors' adherence to them... 4 ‘ 


. 
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= 4 
‘Sites Selected for vs Study 


School of Applied Aerospace ciences, Chanute ‘ 
Air Force RBase,:\Illinois (CHA) 


U.S. Army Ordnance Center and School, Aberdeen * 
‘Proving Ground, Marylljand (ABE) — 


pene of Health Care Science . sheppard Air Force.. 
Base, Texas Sissi ' 


- lk - i ; ’ 
Navy Personnel Research and Development Center, | 
San Diego, alifornia i . 


1. NTC - Navy Training | ‘Center (San Diego) hoe 


2. N.I. = Navy Training! Center (North Tsland) 
' A r d es a. . % 
Naval Training Equipment. center, | : .*s 


| * © : 
| Orlando, Florida ORL) \ ” 


\ 


\ 


pieaal into Maxwell Air Force Base, « 
Alabama (MAX) & 


—_ 


* 


, , sine \ 
; Army Research Institute, feet D.C. (ARI) id ; 
; ~f ina * : 

Orlando, and Maxwell were no “ 
ae six 


/ Of. the above sites, Aberdeen, 
Longer subscribing to PLATO service at the time of the 
;. preparation of this report. ; \ 
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Character 
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Object 


Defines 
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Comments | 


-Restarts/ 
Lesson | 
ax | 


_ Reserved b 
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APPENDIX II 


ry ~ 


‘Strings used as Targets for Computer-Scanning 
| 


Z Character Strings 


| 3 define, n2,\ v2, ne2, ve2 


3) ey * 


restart \ = 


4 . / 


o 


judged, clock, aokist, ntries, jcount, 


words ansent i. 

ae ees 
Anticipated/ wrong, no, arrow, answer, help, 
unanticipated wrongv, ansv, ansu — 
responses ¢ ‘ta ' 
a .| 


note that searches for the string "n2" will also result 


in matches 


nO Pe oO Ca 


located toa 


n10-n19, plus n10@-n150). 


eas n20-n29 Variables, 


The rationale for this 
et variables is as fpllows: n1 would have 
many variables for the purpose of this study (nl), 


given the. ARPA authors suggested that the variables” ni-n9 be 


used as "scratch" variables which would not need to be -define-d. 
require only ,two keystr kes, that set is preferred. ) 


__ (Since the 


The .last consideration was that we needed a'set of variables 
that most people would be likely to have used (hence n80-n89, 


etc. were 


* 


ii selected ‘on: the assumption that lower-numbered 


* 
“ih 
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§ 


Also, the number of authors using a 
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variables would have bee used first). Considering’these 
criteria, the best choice was n2 plus n20-n29. For complete- 
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search. 
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number of times student responses 
to questions have been judged 


amount of time spent in lesson 
or section of lesson 


number of questions answered: 

correctly on first try 
a ae a Se aay i er 

number of attempts ‘at “current at 

question . 

number of internal 6-bit character’ 

codes in student response ’ . 


number of answer-judging commands” re 
encountered before matching the 


